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EMOTION DETECTION IN CUSTOMER 
SERVICE: A CNN-BASED APPROACH TO 
EMPLOYEE ENGAGEMENT 
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Annotatsiya: Ushbu maqolada konvolyutsion neyron tarmoqlari (CNN) yordamida xodimlarning mijozlar bilan muloqoti 
davomida ularning hissiyotlarini aniqlash, shuningdek, deep learning texnologiyasini marketing va inson resurslarini 
boshqarishda qoʻllash masalalari oʻrganiladi. Tadqiqotda hissiy tahlil xodimlarning samaradorligi va mijozlar qoniqishiga 
qanday ta’sir koʻrsatishi ta'kidlanadi. Ma'lumotlar yigʻilib, oldindan qayta ishlangan va CNN modelini oʻrgatish uchun 
ishlatilgan, ammo ma'lumotlar to‘plamining cheklangan hajmi tufayli past aniqlik kabi qiyinchiliklar aniqlangan. Tadqiqotda 
model samaradorligini oshirish uchun individual oʻqitish yondashuvlarini taklif qiladi. Natijalar chuqur oʻrganishni biznes 
jarayonlariga integratsiya qilish mijozlarga yoʻnaltirilgan strategiyalarni va xodimlarni rag‘batlantirish tizimlarini yaxshilashi 
mumkinligini koʻrsatadi. 


Kalit soʻzlar: Deep learning, CNN, hissiyotlarni aniqlash, xodimlar samaradorligi, mijozlar bilan muloqot. 


Abstract: This paper explores the use of Convolutional Neural Networks (CNNs) in detecting employee emotions during 
customer interactions, focusing on the application of deep learning in marketing and human resource management. The 
research emphasizes how emotional analysis can provide insights into employee performance and customer satisfaction. 
Data was collected, preprocessed, and used to train a CNN model, though challenges such as low accuracy due to limited 
dataset size were identified. The study proposes solutions to improve model performance, including individualized training 
approaches. The findings suggest that integrating deep learning into business processes could enhance customer-centric 
strategies and employee incentive systems. 


Key words: Deep learning, CNN, Emotion Detection, Employee performance, Customer interaction. 


AHHoTana: B gaHHoñ pabote ncenegyeTca NCNONb3OBAHNE CBEpTOYHbIX HEPOHHbIX ceTeň (CNN) gna o6Hapyxxenna 
ƏMOĻMÄ COTPYAHNKOB BO BpeMA B3aMMOAeNCTBUA C KNNEHTAMN, C AaKLIEHTOM Ha npumeHeHne rnyOokoro o6yyeHna B 
MapketnHre n ynpaBneHun nepcoHanom. ViccnegoBaHve nogyepkuBaeT, KAK SMOLIMOHANbHbIN aHann3 MOXeT DaTb 
NpeACTAaBNeHNUe O NPOU3ZBOANTENBHOCTH COTPYAHUKOB N YAOBNETBOPeEHHOCTY KNNEHTOB. Bbinu coOpaHbi, npegBapuTenbHO 
oO6paboTaHb! AaHHble n ncnonb3oBaHbi Ana ObyYeHuA mogenu CNN, xota binn BbIABNeHbI TAKNE NpoOnembl, KaK 
HU3KaA TOUHOCTb U3-3a OrpaHnyeHHOoro OObema AAaHHbIX. B uccnegoBaHuu npeANoxKeHb!I peenaa ANA ynyyweHua 
NPpON3BOAUVTeNbHOCTU MOgeny, BKNOYAA MHANBMAYANNZNpPOBAHHbIEe NOAXOAbI K O6yYeHuio. Pe3ynbTaTbl NOKA3bIBAIHOT, YTO 
nHTerpaųna rny6okoro o6yyeHna B OU3HEC-NPOLIECCbI MOXKET YNYYLUUTb CTpaTernn, OpNEHTNpOBAHHbIe Ha KNNEHTOB, M 
CUCTEMbI CTNMYNNpOBaHNA COTPYAHUKOB. 


Knrouesble cnosa: Deep learning, CNN, o6HapyxeHne 3MOLMĂ, NPONSBOAUTENbHOCTb COTPYGHUKOB, B3ANMOJEŇCTBNE C 
KnVeHTamnu. 


INTRODUCTION 


Deep learning, a subset of machine learning, has seen remarkable growth in recent years, transforming 
industries and business practices. This rapid development is driven by advances in computational power and 
the availability of large datasets, enabling deep learning models such as Convolutional Neural Networks (CNNs) 
to excel in complex tasks like image recognition, natural language processing, and emotion detection. CNNs, 
in particular, stand out due to their ability to learn hierarchical representation from data, making them ideal for 
capturing intricate patterns in both structured and unstructured information. 

The business world has begun to recognize the value of deep learning, particularly in enhancing customer 
service and employee performance. In customer-facing industries, maintaining positive interactions is crucial 
for business success. Understanding the emotional state of employees during these interactions can offer 
valuable insights into customer satisfaction and employee engagement. 

In this paper, we propose leveraging CNNs to detect employee emotions in real-time as they communicate 
with customers. The goal is to analyze the proportion of time employees exhibit positive emotions, such as 
happiness and satisfaction, versus negative emotions, such as frustration or anger. By implementing such a 
system, businesses can encourage positive behavior through incentives like performance-based bonuses or 
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salary adjustments, ultimately fostering a more customer-centric environment. This approach offers a novel way 
to blend deep learning with human resources management, optimizing employee performance and improving 
customer experience simultaneously. 

Neural Network 

2.1 Single Neuron —Perceptron A perceptron, the fundamental unit of artificial neural networks, processing 


inputs through a weighted sum and an activation function: finn (Œ; W, b) = o(W : x + b) 
where is the weight vector, is the bias, and represents activation functions like Sigmoid, Hyperbolic Tangent, 
and ReLU. These functions enable non-linear modeling but require iterative weight optimization via gradient 


w ew- =— 
descent: : dw 
While effective for linear data, perceptrons struggle with non-linear patterns, prompting the use of multilayer 
networks. 
2.2 Deep Neural Networks Deep neural networks (DNNs) consist of multiple layers, enhancing the modeling 
of complex non-linear functions. They can be represented as: 


fonn(x) = finn (finn C finn (x) -)) (k layers), 


comprising input, output, and hidden layers. 
2.3 Weight Optimization Weight optimization aims to minimize the loss function: 


While simple perceptrons allow for convex optimization, deep neural networks face significant challenges 
due to the vast number of parameters involved. A common strategy is to employ gradient-based methods, 
where the parameters are adjusted based on the gradients of the loss L with respect to the weights. The 
backpropagation algorithm enhances efficiency by calculating gradients layer by layer, leveraging previously 
computed values via the chain rule. 

2.4 Regularization To prevent overfitting, regularization techniques are crucial, including: 

(1) Weight Decay: Adds a penalty for large weights. 

(2) Dropout: Randomly deactivates neurons during training. 

(3) Batch Normalization: Normalizes layer outputs for faster training. 

These methods collectively improve model generalization and robustness. 

2.5 Convolutional Neural Networks Convolutional Neural Networks (CNNs) are specialized for processing 
grid-like data, such as images, by exploiting spatial dependencies among neighboring pixels. The architecture 
consists of multiple layers that capture features at various levels: lower layers identify simple patterns (e.g., 
edges), while higher layers combine these patterns to form complex representations. 

The key operation in CNNs is convolution, where a filter or kernel is applied to the input data to extract 
relevant features. This is mathematically defined as: 
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with zero-padding to maintain valid boundaries. The extracted features are then fed into fully connected 
layers for classification. This capability makes CNNs particularly effective for tasks such as emotion detection 
during employee-customer interactions. 

Model Development and Data Collection 

3.1 Convolutional Neural Network Architecture To develop our model, we utilized TensorFlow, which 
provides an accessible interface through the Keras API. This framework simplifies the process of building and 
training deep learning models. For our application, we employed a Sequential model, allowing us to stack 
layers in a linear fashion, making it suitable for our Convolutional Neural Network (CNN) architecture. 

3.2 Data Collection and Preprocessing Data collection is a critical step in training our model. We 
gathered images from various online sources, categorizing them into distinct folders based on emotional 
expressions such as “positive _emotions”, “negative _emotions” and “neutral”. This organization facilitates the 
training process by ensuring that each class is adequately represented. 

Following data collection, we performed several preprocessing steps to prepare the images for training. 
These steps included resizing the images to a uniform dimension, normalizing pixel values, and augmenting the 
dataset to enhance the model’s robustness against overfitting. In total, we compiled a dataset of 7,688 images 
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for training and 849 images for testing. The performance of our model was evaluated, with results summarized 
in the following figures, illustrating the accuracy and loss function throughout the training process. 
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3.3 Challenges and Strategies for Improving Accuracy The low training accuracy observed in our model 
can be attributed to several factors. Firstly, the limited size of the dataset may not adequately represent the 
diversity of emotional expressions and variations among individuals. While we collected a total of 7,688 training 
images, this number may still be insufficient for the model to generalize effectively across different scenarios 
and employee characteristics. Additionally, variability in image quality, lighting conditions, and backgrounds 
could further contribute to the model’s inability to learn robust features. 

To improve accuracy, it would be beneficial to train the model for each individual employee. This approach 
can enhance the model’s ability to recognize subtle emotional expressions. By setting up the camera to capture 
images from a consistent angle and background for each employee, we can ensure that the model learns to 
extract features specific to their expressions, thereby improving recognition accuracy. 


SUMMARY 


This study investigates the use of Convolutional Neural Networks (CNNs) to detect employee emotions 
during customer interactions, aiming to enhance both employee performance and customer satisfaction. We 
developed a CNN model using TensorFlow and the Keras API, supported by a diverse dataset categorized by 
emotional states. 

While the model demonstrated functional capabilities, challenges in accuracy were noted, primarily due 
to limited data representation and variability in image quality. To address these issues, we proposed targeted 
training for individual employees to improve recognition of subtle emotional expressions. 

The findings highlight the potential of deep learning to transform marketing and human resource 
management by enabling businesses to foster a more customer-centric environment and develop effective 
incentive programs based on emotional engagement. 
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